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ABSTRACT 

. . la detraraining how reliable is reliable enough and 

how auch error can be tolerated in criterion-referenced testing, the 
following relationships holds (1) the acre specific an objectiv4 is. 
the fewer the iteas required to reliably aeasnre it; (2) the aore 
specific the objectives are, the aore objectives required to cover a 
given span of the curriculua; and (3) the aore specific the 
objectives are, the aore diagnostic the test is in teras of 
uncovering specific deficiencies in a student's achieveaent. There is 
also a four-way interdependency between coaprehensiveness. 

;S!f2nJ!r* 5®!?*^^^^*!' length, and curriculua span. Data are 

presented and discussed with particular eaphasis on the reliability 
problea and specificity- generality diaension of behaviorally stated 
objectives. Data were collected on the Prescriptive Batheaatics 
Inventory and the Prescriptive Reading Inventory. It is concluded 
that aore inforaation is obtained about a student's specific 

weaknesses by aeasuring a large nuaber of specific 
objectives with few iteas than by aeasuring a few general objectives 
with a large nuaber of iteas. (Author/RC) 
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A criterion-referenced test is a test based on a set of specific 
learner objectives stated in terms of the behavioral changes to be escpected 
in the examinee as a result of instruction toward these objectives. If 
you wish to make the objectives the basis for an objective test with gen- 
eral applicability that covers a reasonable span of the curriculum in a 
given content area, then a number of difficttlt prd>leos arise: (1) how 
specific should the objectives be; (2) how important is a given objective 
to achievement in the subject area; (3) how can the objectives be made 
measurable with a paper and pencil test; (4) how many test items should 
be written to measure each objective; and (5) what is an acceptable 
standard of performance? With respect to the total set of objectives 
to be Included in the test, comprehensive coverage of the span of curric- 
ulum desired and the structure or organization of the objectives will also 
be of concern. All these problems need to be solved, and in the end the 
final product must not be unreasonably long or time-consuming to administer. 
These problems are, of course, interrelated. 

An item on a paper and pencil test, when completed by a student, is 
a sai^le of that student's behavior. Furthermore, the item is a sample 
selected from some universe of all possible behaviors which might have 
been selected to represent or measure sor^ particular domain of behaviors. 
In a very real sense, the objective that the item is written to measure 
defines this universe of possible behaviors. The more limited the universe, 
the easier it is to select behaviors to represent it, and the more confi- 
dence we can have that the selected behaviors do, in fact, represent the 
domain* 
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The difficulty of establishing an acceptable standard of perform- 
ance is directly related to the general problem of reliability and validity 
of measurement, which in turn concerns the extent to which the domain of 
behaviors specified by the objective is sampled. There are, then, two 
basic considerations. First, how many instances of the behavior ccasti- 
tutes a reliable measure of the domair; and second, of these, how many 
must a student perform correctly in order to be considered proficient 
in that class of behaviors? The generality-specificity dimension of 
behaviorally stated objectives is directly related to these considerations. 

If tha assessment device is an objective test, perhaps scored by 
coi^uter, then the only information available is correct versus incorrect 
responses to individual items. If we assume some Platonic truth about 
the student with respect to his ability to perform the behaviors described 
by an objective and also assume a level of specificity of the objective 
such that (ideally) if a student can perform correctly one behavior from 
the domain, then he can perform them all, then for any given item on a 
test there are four possible outcomes: (1) the student cannot perform 
the behavior and does not set the item correct; (2) the student cannot per- 
form the behavior but does get the item correct; (3) the student can 
perform the behavior but does not get the item correct; and (4) the 
student can perform the heavier and does get the Item correct. If a 
large number of students were repeatedly tested with items san^ling the 
domain of behaviors specified by such an objective, the proportion of 
correct responses of individual students should approach a stable bimodal 
distribution. The lower mode would give the mean proportion for students 
who cannot perform the behavior and the upper mode wou^d^ive the mean 



proportion for students who can perform the behavior. 

The assunptions above are never true. We have Ignored partial 
knowledge, and no objective is so specific that there is no variation 
in the difficulty of the behaviors it specifies. These two effects 
are confounded in any testing situation. There are no reliable measures. 
The questions then become how reliable is reliable enough and how nrnch 
error of each kind are we willing to tolerate? The following relation- 
ships hold: (1) the more specific an objective is, the fewer the items 
required to reliably measure it; (2) the more specific the objectives are, 
the nwre objectives required to cover a given span of the curriculum; and 
(3) the more specific the objectives are, tht ere "diagnostic" the test 
is in terms of tmcovering specific deficiencies in a student's achieve- 
ment. There is also a four-way interdependenwy between comprehensiveness, 
reliability of measurement, test length, and curriculum span. In develop^ 
ing a criterion-referenced test for general use, all of these elements 
must be balanced and compromises made so that the resulting instrument is 
usable and useful. 

All of the problems mentioned here cannot be discussed at length in 
this short paper* In what follcn^s, data will le presented and discussed 
with particular emphasis on the reliability problem and the specificity- 
generality dimension of behaviorally stated objectives. The data were 
collected on the ppesaHptive Mathematics Inventory (PMI) by John Gessel, 
published by CTB/McGraw-Hill, and the Prssariptive Reading Imentopy (PRI), 
which is to be published later this year. 

The author of the PMI, John Gessel, made an early decision to use 
a single item to measure an objective. In doing this, he was sacrificing 



reliability of measurement in order to isaintain a desired level of 
specificity of the objectives and a desired comprehensiveness and span 
of the curticulum. The decision was deliberate in order to ipake the 
inforroation provided by the test as specific, and therefore diagnostic, 
as possible. We have collected and analyzed data on the PMI in order 
to deten f ne the effect of using what a measurement theorist would 
call a "ot»e-iten» test" to measure an objective. Logically, there are 
reasons to believe that the use of one item is not as serious a problem 
in the PMI as it appears on the surface. There are two ways in which 
an item can give erroneous information: the first is the "false posi- 
tive," in which the student answers an item correctly, but has not 
mastered the underlying behavior; and the second is the "false negative," 
in which the student misses the iten even though he has in fact mastered 
the underlying behavior. 

In the PMI, the false positive response, i^ich generally involves 
guessing, is virtually eliminated ia that the format demands a con- 
structed response which is then encoded in a grid for machine scoring. 
Figure 1 (Appendix A, p. A-1) shows an example item from the PMI and its 
associated response grid. The chance factor for c»st items is quite 
small, 1/100 or less in most cases. There are some itoss where the 
chance factor is 1/10 and a few where it is as large as 1/5, but even 
here the set of the student in taking this test is such that he is un- 
likeljf to mark the grid at random. Note that there are 1,000 ways to 
mark this grid. If the student marks the correct answer in the grid, then 
we can be reasonably sure that he knows the answer. 



The false negative response is more difficult to handle. As long 
as the total test is seen simply as a large collection of one-item 
tests, then the value of the information provided will be suspect. In 
order to justify the use of one it^ per objective, ve must look beyond 
the single item and the particular objective it purports to measure and 
consider the structure and organization of the test as a whole. The 
objectives n^asured by the test and the items written to measure thorn 
are generally hierarchically ordered from less to more complex, find the 
obtained item difficulties confirm this organization within major cate- 
gories. In interpreting the results of testing, the user is urged to 
look at the pattern of right and wrong responses and to discount isolated 
wrong responses when they do not fit into an overall hierarchical pattern. 
Figure 2 (Appendix A, p. A-1) shows a section of the PMI Diagnostic Matrix, 
which is one of the reports returned to the student and his teacher. Notice 
the hierarchy implicit in the arrangement of the report and the pattern of 
responses of the hypothetical student represented in the illustration. 
There are several objectives (items) missed by the student that could pro- 
bably be ignored by him and his teacher. 

If you were constructing a norm-referenced test to a^asure, say, 
arithmetic confutation, you would probably write a comprehensive rationale 
describing the kind of test you want, the kind of items you want, and perhaps 
you would provide a few sample items to give to your item writers. After 
the items were written and tryout data obtained, you would choose items 
whose item difficulty level hovers around .5, because they discriminate 
best, and which are internally consistent. The assumption is that there 
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Is an underlying continuua called "arithmetic confutation achievea^nt" 
and you wish to place students on this continuun as accurately as possi- 
ble. Furthermore, you would want the scores obtain^ fron your test, 
vhen administered to the target population, to be distributed approxl- 
nateXy as the normal distribution. Such test scores have valuable 
psychmitetrlc properties: t^ correlate well with other similar scores, 
provide meaningful derived scores, and so on. 

In constructing a criterion-referenced test, the itraas written to 
me^aure a single, specific objective in that test do not share the 
desirable properties described for a norm-referenced test. In a criterion- 
referenced test it is not necessarily desirable for tb<% Item difficulties 
to be near .5, nor is Internal consistency especially i^ortant. In 
particular, a normal distribution of the objective scores Is distinctly 
not t^anted. There is no assumption of an underlying continuum of achieve^* 
ment on a particular objective, but rather there is the simple assumption 
that each student either has mastered the objective or that he has not 
mastered the objective. The desired distribution of scores on the one 
objective when administered to the target population is bimodal or U- 
shaped as shown in the hypothetical distribution In Figure 3 (Appendix A, 
p. A-2). Desirable hypothetical distributions for an easy test and a 
hard test are shcKim in Figures 4. and 5 (Appendix A, p. A-2), respectively, 
and a hypothetical distribution where learning has apparently occurred 
during the testing session is shown in Figure 6 (Appendix A, p. A-2). 
Examples of each of these distributions based on real data will be d<s- 
scrlbed shortly. 



The PHI reliability study from which these examples were derived 
was carried out last year in two school districts, one in northern, the 
other in southern California. In these schools, we administered Level B 
of the PMI to seventh graders along with a set of criterion tests, and we 
administered Level C of the PMI to eighth graders along with another set 
of criterion tests. To construct the criterion tests, we first selected 
40 items from the PMI to represent the range of material covered by all 
levels of the test. We then wrote alternate forms of these Items to form 
either 5-item or 20-ite» criterion nests to measure each of the 40 objectives 
represented by the selected PMI items. This selection of items was done from 
the tryout version of the test and the final version omitted some of these 
40 items. We also discarded results for two additional tests for which less 
than 10 students correctly answered the corresponding PMI item. The results 
reported here, then, represent 32 rather than 40 objectives from Levels B and 
C of the PMI. The list of objectives a^asured by these criterion tests is 
shown in Table 1 (Appendix B, p. B-1). Summary statistics for these cri- 
terion tests are given in Table 2 (Appendix B, p. B-4). Also included in 
the table is the povnt biserial correlation between the criterion test 
score and its corresponding PMI item. Notice that this correlation tends 
to fall somewnat below the lowest point biserial for any item in the criter- 
ion test itself, though in a few cases it falls within the range. The wsek 
separation between the administration of the P>II item and the criterion tests 
may account for this difference. 

Now look at Figure 7 (Appendix A, p. A-3). This figure shows the 
obtained distribution for criterion test B-2, a 20-ltem test. The solid 
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line represents the total distribution, the dashed line represents the 
distribution of the criterion test scores for those students who correctly 
answered the corresponding PMI item, and the hatched line represents the 
distribution for those students vri.o incorrectly answered the corresponting 
PMI itaa. This distribution do-tely reseables the hypothetical distribu- 
tion for an appropriate test shown in Figure 3 (Appendix A, p. A-2). 
Another exflmple of the sane form of distribution, this tin^ for a 5-itaa 
criterion test, is shown in Figure 8 (Appendix A, p. A-4). Figures 9 
(Appendix A, p. A-5) and 10 (Appendix A, p. A-6) show distributions for 
20-ltem and 5-itein criterion tests, respectively, following the pattern 
for the hypothetical easy test shown in Figure A (Appendix A, p. A-2). 
Figures 11 (Appendix A, p. A-7) and 12 (Appendix A, p. A-8) show the 
20^1tem and 5--ltem criterion test distribution following the hypothetical 
distribution for a difficult test sho\-n in Figure 5 (Appendix A, p. A-2). 
And, finally, Flgurfe 13 (Appendix A, p. A-9) and 14 (Appendix A, p. A-10) 
show 20-item and S-ite-n criterion test distributions conforming to the 
hypothetical distribution shown in Figure 6 (Appendix A, p. A-2) where 
learning has occurred for some students. All of the criterion test distri- 
butions can be classified into one or another of these hypothetical forros 
except for one, B-4, which could not be classified. The classifications 
of all the tests are shown in Table 2 (Appendix B, B-4), alm$ with 
the statistics mentioned earlier. 

Figure 14 (Appendix A, p. A-10) is especially Interesting, since it 
shows a considerable amount of what we have called learning by students 
who missed the corresponding PMI item for this objective. Figure 15 
(Appendix A, p. A-11) gives an example of the kind of item used in this 



test and Its response grid. Note that the three digits requir«>d for 
the answer are found by working from right to left in the proble» and 
that the response grid Indicates that they are to be recorded in the 
right to left order. In the PMI itself, about 8% of the students 
gridded their response in reverse order. Approximately the same per- 
centage of students gridded in reverse order for the criterion test items. 
Apparently the learning cannot be attributed to increased facility in 
gridding the responses. r» item difficulty of the corresponding PMI 
item, however, is substantially lower than the difficulties of any of the 
criterion test items, about .46 as opposed to .63 to .66. The only ob- 
vious difference in the original item was that the size of the blanks 
(square, triangle, etc.) was larger in the PMI item. Our tentative 
conclusion is that, in the process of trying 212 PMI items one week and 
196 similar criterion test items the next week, a substantial number of 
students, in fact, learned how to work the problem. 

In a second phase of the PMI reliability study* we postulated that 
a better estimate of the mastery of npeclfic objectives could be obtained 
by using more of the infort^tlon contained in the total test. Since the 
objectives measured are not Independent of each other, we designated 
subscores for the PMI, and using a stepwise regression procedure we used 
these subscores to predict each of the criterion test scores in turn. 
Table 3 (Appendix B, p. B-5 ff.) gives a short description of each of the 
predictor sub&cores defined for this study and the number of items con- 
tained in each of them. The scores designated "B" were used to predict 
the Level B criterion tests and those designated "C" were used for the 
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Level C criCarion tests* The stepwise procedure was terminated when 
the next predictor to be added would increment the multiple correlation 
by less than .003* In every case, using all of the predictors would 
not have increased the multiple correlation by more than about .01, and 
in most cases the shrunken multiple correlation would have decreased. 

A summary of this analysis is ^own in Table 4 (Appendix B, p. B-9), 
giving the number of predictors selected, the order of selection, the 
multiple correlation, and shrunken multiple correlation. For comparison, 
the point biserial correlations of the corresponding PMI items with the 
criterion tests are also shown. In every case except one, the multiple 
correlation is substantially higher than the point biserial with the 
corresponding PMI item. Criterion test B-9, having to do with the inter- 
section of sets, seems not to be predictable from the PMI subscores as 
defined here. Where the point biserial correlation is small, the differ- 
ence between it and the multiple correlation tends to be large, but where 
the point biserial is large, the difference tends to be small. This 
may reflect a ceiling effect and may also be related to how well the partic- 
ular criterion test measures its objective. No cross validation was 
carried out and, of course, the validity coefficients that would be obtained 
would be smaller than the umltiple correlations shown. 

In mos^ cases the single it^ seems to give a good Indication of 
mastery or non-mastery of its objective. Where the point biserial correla- 
tions are low, the cause can generally be traced to either a somewhat poor 
set of it^ in the criterion test, or to those tests in which learning effects 
seem to be present. The substantial added effort and cost of obtaining regres- 
sion estimates for all of the objectives in the PHI would probably not 
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substantially improve the reliability of the infonnation provided by the 
instrument. 

The PRI reliability study was designed and carried out in a some- 
what different way than was the PMI study. Separate criterion tests were 
not written for the PRI, Instead » as part of the item tryout study , ten 
or more items were written to measure certain objectives. These objectives 
were designated as "reliability study objectives" or PRI criterion tests. 
In some cases » similar objectives were combined into one in order to get 
the needed number of items for this study. The list of PRI criterion 
tests is shown in Table 5 (Appendix B» p« B-10 ff.)« They are designated 
Ay By Cy or D in accordance with the level of the t^t in which they appear. 
This study was further complicated by the large number of items to be tried 
out. Each level of the test was split into several booklets: Level A into 
eight booklets y and Levels and D into four. Each student in the 

study took only one of the 20 separate booklets. 

In most cases y the distributions for the objective scores from the 
PRI did not show the desired U-shaped ct' ve that we found with the PMI 
criterion tests. Of approximately 400 obtained distributions » only 19 
clearly showed the U-shaped form. An additional 16 showed some Indica^ 
tion of the U-shaped distribution^ but most of the distributions either 
approximated a normal distribution- or were more or less severely skewed 
to the left or right* In an effort to measure as purely as possible the 
defined behaviors appearing in the test, we kept stringent control over 
the vocabulary level* As a result ^ the test was generally easy for the 
students taking it, and for this reason more than half of the distribu- 
tions were classified as strongly sk^ed to the right. 
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Figure 16 (Appendix A, p. A-11) shows our best example of a U- 
shaped distribution from the PRI. This objective had only five items 
in the test booklet in which it appeared, so it was not part of the 
reliability study. It does look remarkably like some of the distribu- 
tions found for the PMI criterion tests, for example. Figure 8 (Appendix A, 
p. A-4). Note that it is displaced upward, probably reflecting the fact 
that these are multiple choice items and th^ guessing factor is in evidence. 

The intent of the PRI reliability study was to discover the minimum 
number of multiple choice items that could reasonably be used to measure 
an objective. The procedure t^ed was to select two items at random from 
the pool of ten or more items for that criterion test objective, obtain 
the product moment correlation between the score on those two items and 
the score on the remaining items, and obtain the joint frequency distri- 
bution between the two scores. A third item was then chosen at random 
and the correlation and joint frequency distribution re-computed; then 
a fourth Item was added, and so on until either a correlation of .80 
was obtained or until half of the available items were chosen. 

The results of this analysis show that in most cases, three or 
four items are sufficient to give a reasonably good estimate of mastery 
or non-mastery of the objective. Figure 17 (Appendix A, p. A-12) shows 
the best example where three items are used from a pool of thirteen and 
where the criterion of mastery is specified as two of the three items 
correct. This distribution follows the pattern of an easy test given 
in the hypothetical example shown in Figure A (Appendix A, p. A-2). 
Those students meeting the criterion of two out of three items account 
for the high scores, and those students not meeting the criterion account 
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for the low scores. The higher scoring students who did not meet the 
criterion can be accounted for by the multiple choice format of the 
test* For example^ 14 students would be expected to score four on the 
test by chance t while 18 actually did« On the other hand, less than 
the expected number of students not meeting the criterion scored three 
on the test. 

Figure 18 (Appendix A, p. A- 13) sho\« an exaiuple of one of the 
poorer distributions. Here three items are again used and the criterion 
specified is two of the three correct. The total ninaber of items in the 
pool was 12 9 80 the distribution is for a nine-it^ test. Even here the 
high scores are again accounted for by those students meeting the crlter*- 
ion and the low scores by the students not meeting the criterion , though 
the pattern is not quite as clear. 

Table 6 (Appendix B, p. B-15 ff.) gives a summary of the PRI criterion 
test statistics and the results of correlating the subsets of items 
chosen at random with the remaining items for each test. The reading tests 
are not as homogeneous as the math tests, reflecting the greater generality 
of the reading objectives. In many cases, however, the correlations for 
three or four iten^ with the remaining items for a criterion test objective 
are respectable and it appears quite practical to use three or four items 
to measure these objectives. 

As in the PHI reliability study, a second phase of the PRI study 
was carried out in which a stepwise regression procedure was applied to 
PRI subscores to predict the criterion test scores. This analysis is not 
as informative as the PMI analysis because of the separation of the test 
into many booklets. As a result, there are few predictors available for 
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any one criterion test. The set of predictors arranged bv the criterion 
tests to which they apply are described in Table 7 (Appendix B, p. B-18 ff.). 
A sunnnary of the regression analysis results are shown in Table 8 (Appendix 
B, p. B-21) giving the number of predictors selected, the order of selec- 
tion, the multiple correlation, and the shrunken multiple correlation. 
For comparison, the 4-ltem test correlations with the remaining items 
in the test are also shown. The multiple correlations are somewhat highor 
than the 4-item correlations, but on cross validation the validity 
coefficients would again be smaller. Little advantage is shown here for 
the regression approach. The PRI reliability study will be replicated this 
fall as part of a larger study, so the results reported here can be taken 
as tentative. 

It is our conclusion that more information is obtained about a 
student's specific strengths and weaknesses in math or reading by measur- 
ing a large number of specific objectives with few items than by measuring 
a few general objectives with a large number of items, even though the 
statistical reliability of such scores is lower than one would want. In 
interpreting the results obtained from the PRI, as with the PMI, we will 
urge the student and his teacher to look at the overall pattern of scores 
displayed on a diagnostic map of reading objectives. Practically, 
although the individual indications of mastery or non-omstery of the 
objectives are fallible, few wrong decisions will be made about any 
student when the total pattern of responses is taken into account. 
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Figure 1. EsMsple of a PMX Xt«a aad Its Response Grid. 
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Figure 15. fixra^le of tl» Kind of Item Deed in Criterion 
Test B*19. 




Figure 16 . Distribution of Scores on mi Objective 12 Le^l ht 
PHONETIC PARTS. 
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B-1 



B-1 



B-4 

B-5 

B-6 
B-8 

B-9 



B-10 



B-11 



TABLE 1 

OBJECTIVES MEASURED BY 
THE 32 CRITERION TESTS CONSTRUCTED FOR Tlii- 
PMI RELIABIT.I'n' STUDY 



Given the expanded form of a four-digit numeral, the student will 
write the numeral. (Expanded notation) 

B-2 The student will be able to add three unlike fractions with 
regrouping. 

B-3 The student will be able to subtract a decimal number with two 

decimal places from another decimal number with two decimal places 
with regrouping. ^ 



Given a number line showing the operation of addition of mixed 
numbers and an appropriate open mathematical sentence, the student 
will complete the mathematical sentence to describe the operation. 

The student will be able to divide a three-digit whole number by a 
two-digit whole number with remainder. 

The student will be able to multiply a proper fraction by a mixed number. 

The student will be able to add quantities of time expressed as hours, 
minutes, and seconds with regrouping. 

Given a mathematical sentence expressing the equality of the product 
of two symbolic quantities with th.. product of two other symbolic 
quantities, the student will be able to express true proportions using 
these four quantities. (Ratio and proportion) 

The student will be able to subtract a three-digit whole number from 
a three-digit whole number with regrouping. 

The student will be able to subtract mixed numbers with regrouping. 



B-12 The student will be able to add two decimal numbers each with two 
decimal places with regrouping. 
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Table 1 Continued 



B-13 The student will be able to multiply a three-digit whole number by a 
two-aigit whole number. ^ 



B-14 



C-2 



C-6 



C-7 



The student will be able to divide a mixed number by a proper 
fraction. *^ 



B-15 The student will be able to multiply a two-digit decimal fraction 
by a two-digit decimal fraction. 

B-16 The student will be able to divide a decimal number with one decimal 
place by a two-digit decimal fraction. 

B-i7 Given an equation in whole numbers with a missing factor, the student 

factor ^^^^ ^"^^^^ '^"^ distributive property in supplying the missing 

B-18 Given a sequence of whole numbers involving multiplication by a constant 
value, the student will be able to supply a missing number from the 
sequence . 

B-19 Given a completed subtraction problem with missing digits in the 

subtrahend and the difference, the student will be able to supply the 
missing digits. ' 

B-20 The student will be able to subtract quantities of liquid expressed as 
quarts and pints with regrouping. 

C-1 The student will be able to subtract mixed numbers with regrouping. 

The student will be able to multiply a decimal number with two decimal 
places by a decimal (number with one decimal place. 

C-4 The student will be able to divide a mixed number by a mixed number. 

The student will be able to compute a given percent of a given whole 
number. (Applications) 

The student will be able to add two one-digit negative integers. 

C-9 Given two finite sets, the student will specify the set that is the 
intersection of these sets. (Intersection) 

C-10 Given an open mathematical sentence Involving division of positive 

and negative integers, the student will be able to supply the missing 
dividend . * 

C-11 Given a set of whole numbers, the student will be able to select those 
which are relatively prime. 
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Table 1 Continued 

The student wilJ be able to specify the greatest possible error of a 
measurement made with a given ruler, "^^^-^ error or a 

^"^^ dilT:r.rt\V^'' ^ "^P' '^'^ ^'"^^"^ ^^^^ b« ^^i^ to compute the 
distance between any two points. 

Given an equation in negative integers with a missing factor, the student 
IddenS! '^'^ " ^'^'^ distributive property in supplying* th^eUMnl^ 

'"'^ vaLerJ^h' nJf J '^^'''^ °f ^^^e symbolic 

values with different numerical exponents. (Ix^garithms) 
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TABLE 3 

DESCRIPTION OF PREDICTORS 
USED IN THE 
PMI REGRESS lOii STUDY 



LEVEL B 



1. Operations on the number line involving addition, subtraction, 
multiplication, and division using whole numbers and mixed 
nionbers. (7 items) 

2. Addition of whole numbers with regrouping. (3 items) 

3. Subtraction of whole numbers with regrouping. (4 items) 

4. Multiplication of whole numbers including a repeated addition 
problem and squaring a number. (12 items) 

5. Division of whole numbers with and without remainders. (9 items) 

6. Addition of positive fractions and mixed numbers with and 
without regrouping. (6 items) 

7. Subtraction of positive fractions and mixed numbers with and 
without regrouping. (5 items) 

8. Multiplication of positive fractions and mixed numbers and 
items on greatest common factor, least coimnon multiple, and 
lowest common denominator. (8 items) 

9. Division of positive fractions and mixed numbers. (8 items) 

10. Addition of decimal numbers with and without regrouping. (6 items) 

11. Subtraction of decimal numbers with and without regrouping. (5 items) 

12. Multiplication of decimal numbers. (8 items) 
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Table 3 Continued 

13. Division of decimal numbers. (10 items) 

14. Addition, subtraction, multiplication, and division of 
negative Integers. (4 items) 

15. Rounding numbers to the nearest 10. 100, or 1,000 and 
estimating answers to addition, subtraction, multiplication, 
and division problems. (5 items) 

16. Commutative and associative properties of addition and multiplication, 
distributive property of multiplication over addition, and the 
identity elements for addition and multiplication. (10 items) 

17. Determining missing elements in sequences of whole numbers or 
positive fractions where the sequence is determined by addition, 
subtraction, multiplication, or division by a constant value. 

(7 items) 

18. Missing addends and factors involving the inverse relationships 
of addition, subtraction, multiplication^^ and division. (4 items) 

19. Metric geometry involving linear measurement, area, volume, 
angles (90©), and line graphs. (9 items) 

20. Concepts of temperature and money. (4 items) 

21. Addition, subtraction, multiplication, and division of lengths, 
money, and liquid measures. (10 items) 

22. Place value from .99999 to 999, 999. 999. expanded notation, 
and exponential notation. (8 items) 

23. Non-base ten systems with conversions and operations and modular 
systems. (13 items) 

24. Non-metric geometry involving recognition and classification of 
geometric forms. (13 items) 

25. Percentage concepts involving conversion of fractions and decimal 
numbers to percentages and computing percentage of a given number, 
(o items) 



26. 



Set concepts Including union, intersection, and Cartesion product, 
(5 items) 
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LE VEL C 



1. Trigonometry involving the definitions of sine, cosine, and 
tangent and an application problem. (4 items) 



2. 



3. 



Operations on the number line involving negative integers. 
(4 items) 

Addition, subtraction, multiplication, division, and squaring 
of whole numbers. (5 items) 



4. Addition, subtraction, multiplication, and division of positive 
fractions and mixed numbers. (16 iten») 

5. Addition, subtraction, multiplication, and division of decimal 
numbers. (4 items) 

6. Addition, subtraction, multiplication, and division of negative 
integers. (4 items) 

7. Addition, subtraction, multiplication, and division of negative 
fractions and mixed numbers. (16 items) 

8. Rounding numbers to the nearest 10, 100, or 1,000 and estimating 
answers to addition, subtraction, multiplication, and division 
problems. (5 items) 

9. Commutative property of addition and muliplication. (5 items) 

10. Associative property of addition and multiplication. (5 items) 

11. Distributive property of multiplication over addition. (3 items) 

12. The identity elements for addition and multiplication. (5 itemn) 

13. Inverse relation of addition with subtraction and multiplication 
with division. (5 items) 

14. Determining missing elements in sequences, some with alternating 
positive and negative numbers, where the sequence is determined 
by addition, subtraction, multlpiication. or division by a 
positive or negative constant value. (4 items) 



15, 



16. 



19 

20, 
21. 

22. 



24 
25 
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Table 3 Continued 



^H^f^? involving the inverse relationships 

of addition, subtraction, multiplication, and division and using 
positive and negative whole numbers and fractions. (14 items) 

Number theory involving prime, composite, and relatively prime 
numbers and odd vs. even numbers. (4 items) 



17. Solution of equations. (4 items) 
18. 



Linear measurement including preclseness, greatest possible error, 
relative error, and scientific notation. (5 items) 

Metric geometry involving linear measurement, area, and volume, 
(o Itet&s) 

Non-metric geometry involving recognition and classification of 
geometric forms. (24 items) 

Percentage concepts involving conversion of fractions and decimal 
numbers to percentages and computing percentage of a given number, 
vo Items} 

Set concepts Including union, intersection, Cartesian product, 
equivalence, subsets, empty set, disjoint sets, cardinality, and 
Venn diagrams. (10 items) 

23. Reading values from various kinds of graphs. (7 items) 

Statistics including finding mean, median, mode, and range. 
(4 items) 

Probability including simple events, coni,i nations, "and," and 
"or" problems. (4 items) 
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TABLE 4 

SUMMARY OF RESULTS FROM THE STEPWISE REGRESSION ANALYSIS PREDICTING THE 32 

PMI CRITERION TESTS FROM SUBSCORES OF THE PMI TEST 
me Stepwise Analysis Was Stopped When the Next Predictor Would Contribute 

Uss Than .003 to the Multiple Correlation 



CRITERION NO. OF 
TEST PREDICTORS 



ORDER OF 
SELECTION 



MULT, R 



SHRUNKEN 
MULT.R 



PMI ITEM 
PT.BIS. 



B-1 


6 


B-2 


4 


B-3 


8 


B-4 


6 


B-5 


5 




4 


B-8 


8 


B-9 


8 


B-IO 


7 




4 


B-12 


7 


B-13 


7 


B-14 


5 




4 


B-16 


6 


o-X/ 


8 


B-18 


6 


B-19 


8 


B-20 


6 


C-1 


6 


C-2 


7 


C-4 


6 


C-6 


7 


C-7 


10 


C-9 


6 


C-IO 


6 


C-11 


9 


C~14 


6 


C-16 


7 


C-18 


9 


C-19 


9 


C-20 


10 



4,26,5,13,3,18 
6,7,5,8 

5,11,8,10,23,1,3,15 
1,7,15,25,19,23 
5,7,4,26,3 
9,8,17,3 

6,17,5,1,2,22,15,18 
23,21,19,16,25,4,6,18 
3,5,15,16,2,20,24 
7,25,8,1, 

10,8,5,21,11,19,26 

5,8,4,7,6,11,3 

9,19,15,11,24 

12,8,15,3,21,17 

13,25,24,18,9,15 

8,16,5,25,15,9,12,4 

8,17,4,7,5,25 

18,5,17,10,3,24,16,4 

13,15,25, 7, •>3,1 

4,7,24,3,17,22 

^,5,3,2,1,8,23 

4,25,9,12,17,7 

7,4,12,10,3,21,20 

4,6,2,14,15,12,1, 

10,11,9 

5,22,2,3,20,4 

7,13,1,6,3,21 

20,16,14,13,6,17, 

25.1,3 

4,24,15,20,25,17 
18,7,21,24,2,3,14 
4,23,12,9,1,2,22,11 
5 

11,22,13,2,1,8,16 
24,12 

7,17,2,23,3,1,16 
25,19,13 
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.73 
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.65 


.64 


.50 
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.59 


.58 
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.68 


.67 


.53 


.75 


.74 


.62 


.63 
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.73 


.72 


.59 


.71 


.70 


.52 
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.51 


.41 


.60 


.59 


.37 


.78 


.77 
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.59 


.60 


.77 


.76 


.61 


.70 


.69 


.64 


.7i 


.70 


.38 


.68 


.67 


.42 


.68 


.67 


.58 
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TABLE 5 

PKI OBJECTIVES MEASURED BY AT LEAST 10 ITEMS IN THE TRYOUT STUDY 
AND USED IN THE PRI RELIABILITY STUDY 



A-1 



A-2 



J^rnf^l"^^"' ""^^^ ^^^^ pronouns in sentences by choosing the 

pron^^ LT°"" ' substituting correct 

pronoun for a noun in a sentence. (10 Items) *-orrecc 

The student will be able to match like or unlike entitles hv 
given words with their definitions or synon^^i US IteL) ' ' ' 

A-3 The student will be able to: 

1. choose the cause of a given effect in a story, 

^ctuL?"^''^°"' "^"^"'"^ '^^^"S inferences about a given 

^' m LT''""^ '^^'"^ inferences about a given 

4. answer questions that require the drawing of conclusions frora a 
given passage. (16 Items) «v.Au»ions rrora a 

A-4 The student will be able to: 

1. choose the cause of a given effect in a story 

TiTtlleT"''"'"" ""'^"S Inferences about a given 

s'^o^^' ir''°" '''' inferences about a given 

4. answer questions that require the drawing of conclusions from a 
given passage. (16 Items) * 

K^i^ ^^^^ '° recognise like vowel sounds and to 
orintof H^'^f ^^""^ "^^^^i^S «ords with 

pIct"esr^Il1^Ll^^^' "^^'^ ^^'^ ^^^"'^^ - P^^"-^ -^<i« With 



B-1 



B-2 



TcL^lfT ""PP^y contracted form and possessive 

form of given verb phrases. (12 Items) F«»Bessive 

B-3 The student will be able to: 

1. match contractions used in sentences with word pairs or choose the 
contraction for a word pair, and 

^ruems) ""'""'^'"'^ '''"^ possessive form of given verb phrases. 



B-11 

Table 5 Continued 



B-4 The student will be able to: 

1. match given singular and plural words with pictures or choose 
designated singular or plural forms ..f words, and 

2. natch contractions used in sentences with word pairs or choose 
the contraction for a word pair. (15 Items) 

B-5 The student will be able to indicate which of certain sentences 

^v^nf^hJ^^ specific story part (e.g.. last part) or choose the 
event that happened before or after a given event. (12 Items) 

B-6 The student will be able to: 

1. answer questions that require the making of inferences about a 
given story. 

2. answer questions that require the drawing of conclusions from a 
given passage. 

3. answer questions that require the prediction of future action from 
a given story, and 

4. choose words or sentences that describe the feelings of a character 
in a given story. (17 Items) 

The student will be able to choose a statement that best represents a 
passage or story or choose the best title for a story. (11 Items) 

B-8 The student will be able to: 

1. answer questions about details given in a story, 

2. name the story or story part in which given events occurred, and 
J. Identify true statements about a given story. (18 Items) 

B-9 The^tudent will be able to indicate which of certain sentences occurred 
in a specific story part (e.g., last part) or chooce the event that 
happened before or after a given event. (10 Items) 

B~10 The student will be able to: 

1. i"^icate the sense to which certain sensory images appeal, indicate 
which of certain sentences or phrases most appeals to a particular 
sense, or choose the example of sensory imagery that answers a given 
question or completes a given sentence, and 

identify descriptive words and phrases from those given, or choose 
the most appropriate descriptive word for a given person or thing. 
(IJ Items) ^ 



B-7 



2. 




B-12 

Table 5 Continued 



B-11 The student will be able to: 

1. draw inferences from story material by choosing responses to 
answer questions or to complete sentences, ^««P<^^«s to 

tit: c'o1cl^\oTtf """^'^^"^ '"^^''^^^ 
3. anticipate or predict probable future actions or outcomes for 
stoxy material by choosing the best answer To glv^n J^tions 
or by completing given sentences. (17 Items) ^"^^tions, 

b'^st'tme'for'i f° '^""'"'^ (lesson), or 

oest title for a given selection. (10 Itena) 

^%hir^?rf;fi^\^^ ^^^^ ^^'^'^^ ^"i'^ to a word 

so that it fits in a given sentence or phrase. (12 Items) 



B-12 
C-1 

C-2 The student will be able to: 



1. choose words or sentences that describe character traits Bttttud.. 

C-3 The student will be able to: 

1. recognize figures of speech by choosing a sentence that has the 

2. locate a simile in a given passage and choose its meaning or identify 

a siSir.' ^r"^^\«- - ««ntence containlTa fl^le" choose 

a simile to define a phrase, or identify a simile. (Il ite^' 

C-4 The student vill be able to: 

1. identify the cause of a given effect appearing in a storv «nH 

2. identify the effect of a given action occur^Jf g'^n^ltTr^ . ^^2 Items) 
C-5 The student will be able to: 

1. answer questions about details given in a story. 

3 "iTrlT T''''^ ^'""^ ^^I'^h given events occurred, and 

3. identify true statements about a given story. (13 Ite^) 

C-6 The student will be able to: 

1. answer questions about details given in a story. 

3* T""^ ^'^'^^^ P^" i" given events occurred, and 

3. identify true statements about a given story. (14 Items) 
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C-7 



C-10 
D-1 



Slaractertr^l'.'" ^^^^ sentences that describe 

TZltll n 1/ attitudes, or manner of speech (e.g.. spoke proudly) 
or answer questions about 4 aracter traits. (11 Itei^) ^ -- ^ 

C-8 The student will be able to: 

1. recognise figures of speech by choosing a sentence that has the 
same meaning as a figure of speech (idiom) appearing in a given 
sentence, © * » « 6j-vcu 

2. locate a simile in a given passage and choose its meaning or 
identify the object it describes, choose a sentence containing 
a simile, choose a simile to define a phrase, cr identify a 
simile, and ^ 

3. choose the definition of a metaphor, complete a given sentence 
with a affitaphor. or identify a metaphor. (18 Items) 

C-9 The student will be able to: 

1. answer questions about details given in a story, 

3* "iZt^f f '''^ ^'^^ ^^^'^^ Si^^" occurred, and 

J. identify true statements about a given story. (10 Items) 

^^t^MM^/^^^ ^^^^ ^° identify the theme, moral (lesson), or 
best title for a given selection. (11 Items) 

The student will be able to identify words th^t are compounds or 
Choose a word to make a compound word. (13 Items) 

D-2 The student will be able to.- 

1. choose the correct definition for given affixed words. 

and"^^ ''''"^''^ definition of the affix in a given affixed word. 

3. choose the correct affix for an unfinished word that will complete 
a sentence. (24 Items) *^ 

D-3 The student will be able to: 

1. recognize figures of speech by choosing a sentence that has the 
same meaning as a figure of speech (idiom) appearing in a given 

2. locate a simile in a given passage and choose its meaning or 
identify the object it describes, choose a sentence containing 
a simile, choose a simile to define a phrase, or identify a 
simile, and ' 

3. choose the definition of a metaphor, complete a given sentence 
with a metaphor, or identify a metaphor. (15 items) 
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B-14 
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1. 

2. 

3. 



D-4 The student will be able to; 

1. answer questions about details given in a story. 

3 r*''^ ^^i^^ 8i^«° events occurred, and 

3. identify true stateinents about a given story, (li Items) 

D-5 The student will be able to: 

draw inferences from story material by choosing responses to 
answer questions or to complete sentences, 

tlZ c^LlusioTan"^'^"^ ^"^^^"^^^^ ^"^^"^^ ^ ^^-^^"^ 
anticipate or predict probable future actions or outcomes for 
story material by choosing the best answer to given qSsUo^ 
or by completing given sentences. (15 Items) ^"escions, 

D-6 The student will be able to: 

recognize figures of speech by choosing a sentence that has the 
IZeTcT' ^ ''^"^^ ^"'^"'^ appearing' in' a^gi^e'^ 

]aT11/ f^""^^?/" ^ ^^''^^ passage and choose its meaning or 
identify the object it describes, choose a sentence con^fining a 

siSie! ^ "^"""^ '° ""^^'"^ ^ ^''^ identify a ^ 

3. choose the definition of a metaphor, complete a given sentence 
With a metaphor, or identify a metaphor. (12 Items) 

D~7 The student will be able to: 

draw inferences from story material by choosing responses to 
answer questions or to complete sente nces. ^P^^ses to 

bls't lllT.lilZlT''' '"^'''^"^ " '^^-^^"^ 

anticipate or predict probable future actions or outcomes for 
story material by choosing the best answer to giv^n q^s^ions 
or by completing given sentences. (11 Items) ^"escions. 



1. 
2. 



1. 
2. 

3. 
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TABLE 7 

DESCRIPTION OF PREDICTORS 
USED IN THE PRI REGRESSION STUDY BY 
APPLICABLE CRITERION TESTS. 



3 
4 



CRITERION TESTS A-l and A-2 

1. Choosing a correct pronoun to cc^plete a sentence or substituting 
a pronoun for a noun. (4 Items) swiwuting 

2. Choosing the correct form of a verb to agree with the subject 
of a sentence. (4 Items) »«Bjecc 

Use of question mark, exclamation point, and quotation marks. (6 Items) 
Combining subjects with predicates to make meaningful sentences. 

5. Match words with their definitions or synonyms. (6 Items) 
CRITERION TEST A-3 

2. Identify character traits, feellngg, and natives; Identify cause and 
CRITERION TEST A-4 

CRITERION TEST B-1 

^* mlte^f ' '^^"^"^ """^ R-controlled vowels. 

r:o^1s%TY!'(r?t:L^"°^'^ substituting consonants, and 

^' ^"^^^h"^ '^h ^yUables in a word, recognise silent letters 

(11 Ue^) ^^^"'^^y °f phon.uic parts of words 
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Table 7 Continued 



4. Choose a correct pronoun to complete a sentence or substitute a 
pronoun for a noun. (4 Items) 

5. Identify spelling changes caused by adding various endings to words. 
(4 Items) 

CRITERION TEST B-2 AND B-3 (COMBINED) 

1. Identify affixes and root words. (6 Items) 

2. Identify contractions with the contracted words. (7 Items) 

3. Distinguish between positive, comparative, and superlative forms of 
adjectives and recognize past, present, and future tense. (7 Items) 

4. Use context to determine the n^aning of words and phrases. (9 Items) 
CRITERION TESTS B-4, B-5, B-6 AND B-7 

1. Match words with their definitions or synonyms. (8 Items) 

2. Identify honwnyms and heteronyms. (6 Items) 

3. Recall story details and sequence of events. (8 Items) 

4. Recognize cause and effect relationships, draw inferences and 
conclusions from a story, and infer future action. (5 Items) 

CRITERION TESTS B-8, B-9, B~10.*«B-11, AND B-12 

1. Recall story details and sequence of events. (6 Items) 

2. Identify character traits, feelings, and motives. (7 Items) 

3. Identify sensory imagery and figurative language in a story. 
(7 Items) 

4. Recognize cause and effect relationships, draw inferences and 
conclusions from stories, and infer future action. (9 items) 

5. Identify the main idea, fantasy, and time of occurrence of a 
story. (7 Items) 



Table 7 Continued 
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CRITERION TEST C-1 

1. Identify silent letters, digraphs and diphthongs, and variant 
phonetic parts In words and blend sounds to make new words. 
(12 Items) 

2. Identify affixes and root words and choose definitions for 
affixes and affixed words. (11 Items) 

3. Use of commas and exclamation marks. (6 Items) 

4. Identify the referent of a pronoun, Identify the meaning of 

given phrases in sentences, and complete sentences with appropriate 
phrases. (12 Items) 

CRITERION TESTS C-2 AND C-4 

1. Recall story details and recognize story setting. (6 Items) 

2. Recognize cause and effect relationships. (4 Items) 

3. Distinguish between real and make believe and recognize fables. 
(4 Items) 

CRITERION TESTS C-6, C-7, C-8, C-9, AND C-10 

1. Recall story details and sequence of events, and recognize the 
setting of a story. (9 Items) 

2. Identify character traits, feelings, and motives and recognize 
sensory Imagery in a story. (4 Items) 

3. Recognize figurative language, simile, and metaphor in a storv. 
(8 Items) ' 

4. Draw conclusions from a story and infer future action. (5 Items) 

5. Recognize cause and effect relationships. (4 Items) 

6. Recognize the j:;ain idea and mood of a story. (6 Items) 

7. Distinguish between real and make believe in a story and recognize 
fables. (5 Items) 
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TABLE 8 

SUMMARY OF THE RESULTS OF THE PRI REGRESSION ANALYSIS 
The Stepwise Analysis Was Stopped When the Next Predictor 
Would Increase the Multiple Correlation By Less Than .003 



CRIT. 


TEST 


NO. OF 
PREP. 


ORDER OF SEL. 


MULT. R 


MULT. R 


4 ITEM 
CORRELATION 


A-2 




3 
3 


1. 5, 2 
• 5. 3, A 


.82 
.81 


.82 
.81 


.48 .65 
.44 .64 


A-3 
A-3 


(Cause) 
(Inf.) 


2 
2 


2, 1 
2, 1 


.69 
.58 


.69 
.58 


.53 .56 
.53 .56 


A-4 




1 


1 (1 pred. only) 


.60 




.28 .35 


B-1 




2 


1. 2 


.76 


.76 


.64 .62 


B-2, 


3 


A 


2, 1, 3, A 


.79 


.79 


.46 .62.53 .53 


B-4 
B-5 
B-6 
B-7 




3 
3 
A 
3 


3. 4. 1 
3, 1, 4 
3, 1, 2, 4 
3, 4, 1 


.74 
.59 
.55 
.67 


.74 
.59 
.54 
. o / 


.61 .51 
.41 .38 
.51 .49 
. .36 .28 


B-8 

B-9 

B-10 

B-11 

B-12 




A 
5 
5 
5 
5 


4, 2, 3, 1 
4, 2, 3, 5, 1 
1. 5, 3, 4, 2 
4, 2, 5, 3, 1 
4, 5, 2, 3, 1 


.83 
.72 
.76 
.70 
.74 


.83 
.71 
.76 
.69 
.74 


.65 .68 
.39 .42 
.53 .59 
.56 .55 
.34 .50 


C-1 




3 


2, 1, 4 


.78 


.78 


.62 .67 .71 


C-2 
C-4 




3 
3 


1, 3, 2 
1, 2, 3 


.76 
.68 


.75 
.68 


.62 .66 .58 
.47 .45 .47 



! !' I* ?• ^' ^' ^ -69 .68 .60 

5 5. 7. 1. 2. 6 .68 .68 .37 .36 .41 

i r ? , -51 .53 .55 .60 

5 1. 7. 4, 5. 6 .72 .72 .58 .60 .60 

5 5, 7, 1, 6, 4 .70 .70 .43 .44 .48 



